Papers with deep networks
Self-supervised Representation Learning for Speech Processing (2022.naacl-tutorials)
Copied to clipboard
Hung-yi Lee, Abdelrahman Mohamed, Shinji Watanabe, Tara Sainath, Karen Livescu, Shang-Wen Li, Shu-wen Yang, Katrin Kirchhoff
| Challenge: | Self-supervised representation learning (SSL) uses proxy supervised learning tasks to obtain training data from unlabeled corpora. |
| Approach: | They propose to survey the latest SSL techniques, tools, datasets, and performance achievement in speech processing to scale up current machine learning technologies. |
| Outcome: | The proposed tutorial is highly relevant to the special theme of ACL about language diversity. |
Emergent Language-Based Coordination In Deep Multi-Agent Systems (2022.emnlp-tutorials)
Copied to clipboard
| Challenge: | Pre-trained deep networks are the standard building blocks of modern AI applications. |
| Approach: | This tutorial will introduce deep net emergent communication and discuss current shortcomings . participants will implement and analyze two emergentic communication setups from the literature . |
| Outcome: | The presentation will cover various topics from the present and recent past, as well as discussing current shortcomings and suggest future directions. |
Compositional Networks Enable Systematic Generalization for Grounded Language Understanding (2021.findings-emnlp)
Copied to clipboard
| Challenge: | a recent study shows that deep networks can mimic some human language abilities when presented with novel sentences . a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains is critical to building safe and fair robots, says a new study. |
| Approach: | They build a general-purpose mechanism that enables agents to generalize their language understanding to compositional domains. |
| Outcome: | a new network generalizes its language understanding to compositional domains while generalizing its knowledge when prior work does not. |
LexSym: Compositionality as Lexical Symmetry (2023.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to generalize compositional models fail to generalise from small datasets. |
| Approach: | They propose a domain-general and model-agnostic formulation of compositionality as a constraint on symmetries of data distributions rather than models. |
| Outcome: | The proposed procedure matches or surpasses state-of-the-art, task-specific models on COGS semantic parsing, SCAN and Alchemy instruction following, and CLEVR-CoGenT visual question answering datasets. |
Class based Influence Functions for Error Detection (2023.acl-short)
Copied to clipboard
Thang Nguyen-Duc, Hoang Thanh-Tung, Quan Hung Tran, Dang Huu-Tien, Hieu Nguyen, Anh T. V. Dau, Nghi Bui
| Challenge: | Influence functions (IFs) are powerful tools for detecting anomalous examples in large scale datasets. |
| Approach: | They propose a method to explain the instability of IFs by leveraging class information to improve the stability of ifs. |
| Outcome: | The proposed method improves performance and stability while incurring no additional computational cost. |
Tensor Product Generation Networks for Deep NLP Modeling (N18-1)
Copied to clipboard
| Challenge: | Using Tensor Product Representations (TPRs) we propose a new architecture for natural language processing based on the principle that hypothesis space for learning includes network hypotheses that are independently known to be suitable for performing the target task. |
| Approach: | They propose a Tensor Product Generation Network (TPGN) which is capable of carrying out TPR computation but uses unconstrained deep learning to design its internal representations. |
| Outcome: | The proposed architecture outperforms baselines on the COCO dataset and can interpret internal representations and operations. |
Did the Model Understand the Question? (P18-1)
Copied to clipboard
| Challenge: | Using the notion of “attribution,” deep learning models often ignore important question terms. |
| Approach: | They propose techniques to analyze the sensitivity of a deep learning model to question words . they use attribution to generate adversarial questions using visual and tabular questions . |
| Outcome: | The proposed techniques reduce the accuracy of a visual question answering model by 61.1% and that of 'tabular' question answering models by 3.3%. |
Learning Deep Transformer Models for Machine Translation (P19-1)
Copied to clipboard
| Challenge: | Neural machine translation models have advanced the previous state-of-the-art by learning mappings between sequences via neural networks and attention mechanisms. |
| Approach: | They propose to use layer normalization to pass the combination of previous layers to the next layer to improve the model. |
| Outcome: | The proposed model outperforms the shallow Transformer-Big/Base baseline model on English-German and Chinese-English tasks by 0.4-2.4 BLEU points. |
Superpose Task-specific Features for Model Merging (2025.emnlp-main)
Copied to clipboard
| Challenge: | Existing methods for model merging are limited by resource demands . recent studies validate the linear representation hypothesis . |
| Approach: | They propose a method that superposes task-specific features from individual models into a merged model. |
| Outcome: | The proposed method outperforms existing methods on multiple benchmarks and models. |
Adversarial Mixing Policy for Relaxing Locally Linear Constraints in Mixup (2021.emnlp-main)
Copied to clipboard
| Challenge: | Experiments on five text classification benchmarks and five backbone models have shown that our methods reduce the error rate over Mixup variants in a significant margin (up to 31.3%), especially in low-resource conditions (upto 17.5%). |
| Approach: | They propose to add a small adversarial perturbation to the mixing coefficients rather than the examples to relax locally linear constraints. |
| Outcome: | Experiments on five text classification benchmarks and five backbone models show that the proposed methods reduce the error rate over Mixup variants by 31.3%, especially in low-resource conditions. |
An Effective Label Noise Model for DNN Text Classification (N19-1)
Copied to clipboard
| Challenge: | Existing methods to train deep neural networks with label noise are limited to image classification models . label noise is important because of the large number of errors and errors in training datasets . |
| Approach: | They propose a non-linear processing layer that models label noise into a convolutional neural network (CNN) they add a noise model layer on top of their target model to account for label noise . |
| Outcome: | The proposed approach is robust to label noise and can learn better sentences . it is based on extensive experiments on text classification datasets . |
EBERT: Efficient BERT Inference with Dynamic Structured Pruning (2021.findings-acl)
Copied to clipboard
| Challenge: | Pruning has been demonstrated as an effective way of reducing computational complexity for deep networks, especially CNNs for computer vision tasks. |
| Approach: | They propose a dynamic structured pruning algorithm that prunes model weights at run-time . they propose to prune the unimportant heads in multi-head self-attention layers . |
| Outcome: | The proposed algorithm outperforms state-of-the-art methods on different tasks. |
Value Residual Learning (2025.acl-long)
Copied to clipboard
| Challenge: | Existing decoder-only transformers fail to preserve initial token-level information in deeper layers. |
| Approach: | They propose a new architecture that incorporates value residual connections in addition to hidden state residuals. |
| Outcome: | The proposed architecture reduces KV cache size by nearly half with only a small performance penalty and can be integrated with other KV-efficient methods. |
Resonant Context Anchoring: Decoupling Attention Routing and Signal Gain at Inference Time (2026.findings-acl)
Copied to clipboard
Mingkuan Zhao, Yide Gao, Wentao Hu, Suquan Chen, Tianchen Huang, Zhenhua An, Zetao Chang, Xiayu Sun, Yuheng Min
| Challenge: | Existing mitigation strategies rely on suppressing specific neuron activations or employing computationally expensive contrastive decoding mechanisms, which often result in increased perplexity or significantly elevated inference latency. |
| Approach: | They propose a lightweight inference-time intervention method grounded in the perspective of residual stream signal dynamics to resolve the signal attenuation of external evidence during its propagation through deep networks. |
| Outcome: | The proposed method improves contextual faithfulness across multiple factual consistency and strong knowledge-conflict tasks while maintaining the model’s general language understanding capabilities. |
Universally Empowering Zeroth-Order Optimization via Adaptive Layer-wise Sampling (2026.findings-acl)
Copied to clipboard
| Challenge: | Existing methods for fine-tuning Large Language Models are slow and lack of performance. |
| Approach: | They propose a Zeroth-Order optimization framework that uses forward passes to fine-tune Large Language Models. |
| Outcome: | The proposed framework achieves 1.7 to 3.0 wall-clock acceleration on LLaMA and OPT models. |